蚂蚁 inclusionAI 开源 PIBench：评测 AI 编程智能体的支付集成基准

inclusionAI; PIBench

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork1 (1); Star26 (26)You must be signed in to star a repository

PIBench is an open-source benchmark for evaluating AI coding agents on realistic, end-to-end payment integration tasks.

Repository files navigation

Alipay-PIBench is a repository-level benchmark proposed by Alipay for evaluating AI coding agents on realistic payment integration tasks. Built around Alipay Open Platform products and business-oriented repositories, it comprises nine product-specific projects and 18 task instances across Basic (Functional Payment Completion) and Advanced (Risk-Aware Payment Hardening) scenarios. The benchmark evaluates functional correctness, reliability, security, and business-state consistency.

Payment Benchmark Construction. Each payment product is paired with a representative project, and each task combines the target product, business workflow, initial repository, and scenario-specific integration request; Rubric-Grounded Evaluation. Scenario-specific rubrics define deterministic checks and LLM-assisted criteria, producing rubric-aligned evidence for structural, executable, and payment-domain requirements; Paired Skill Study. Each agent is evaluated on the same task, project, instruction, and environment with and without alipay-payment-integration, enabling comparisons by product, scenario, agent, and skill condition

🏆 Main Results

Model Capability

† Post-publication result; not included in the paper's main experiments.

Model capability explains much of the observed performance spread and varies across products and scenarios. Under the with-skill condition, overall mean Rubric Pass Rate (RPR) ranges from 68.58% to 92.72%. Basic and Advanced tasks provide complementary views of an agent's ability to construct a payment flow and preserve safety-related payment behavior.

For results and detailed analyses of the remaining research questions, please refer to the paper.

🚀 Quick Start

Requirements

Run the suite on a Linux host or cloud server with:

Bash; Python 3; Git; Docker Engine, with permission for the current user to access the Docker daemon; Network access to the selected model provider, Docker image registry, and task-specific package repositories; Enough disk space to build task images and store run artifacts

The default configuration enables Docker because each task instance defines its own application, database, and development environment.

Run the Benchmark

This release is self-contained. It includes the benchmark suite, deterministic and LLM-assisted evaluations, three agent adapters (Claude Code, OpenClaw, and Hermes), the PaySkills runtime source, Docker orchestration, concurrent execution, and a top-level run.sh. You do not need to install the PaySkills platform.

config/.env is loaded automatically and ignored by Git. Do not put real API keys or Alipay sandbox private keys directly in config/config.yaml.

To keep multiple local configurations, pass one explicitly or set PAYSKILLS_CONFIG:

⚙️ Configuration

The active configuration is config/config.yaml. config/config.example.yaml is the reference template.

Run

parallelism: maximum number of task instances executed concurrently; output_dir: safe relative output directory; do not use package paths such as benchmark_suite/, config/, or src/; timeout_sec: timeout applied separately to task/run.sh and evaluation/evaluate.sh; task_instances: optional list of task instance IDs; an empty list selects all 18

Keep configuration types literal: positive integers for parallelism and timeout_sec, a YAML list for task_instances, mappings for env and runtime_inputs, and booleans for docker.enabled and docker.build.

Agent

type: claude-code, openclaw, or hermes; mode: no-skill or with-skill; model: model name understood by the selected adapter and provider; base_url: provider endpoint; configure it together with api_key_env; api_key_env: name of the environment variable containing the key, not the key itself

Set the referenced variable in config/.env:

LLM Judge

Fill all three judge settings together. Store the referenced key in config/.env:

The agent and judge may reference the same environment variable when they use the same provider credential.

Extra Environment Variables

Values under env are copied into every task instance process:

Use runtime_inputs, not env, for structured files or secrets requested by specific task instances.

Alipay Sandbox Runtime Input

Some task instances require Alipay sandbox information. Obtain it from the Alipay Developer Sandbox, and keep it outside the benchmark suite and Git history.

For identity-sensitive tasks, app_id and seller_id are distinct: the app ID identifies the sandbox application, while the seller ID is the merchant PID receiving funds. Do not copy the app ID into the seller ID field.

To provide sandbox data from config/.env:

Then fill the variables required by the selected task instances in config/.env. This suite contains Java and non-Java projects, so provide both PKCS#8 and PKCS#1 merchant private-key forms for a complete run.

You can instead use a local JSON file:

Relative paths are resolved from the directory containing the selected configuration file. With config/config.yaml, the example above resolves to secrets/alipay-sandbox-keys.json at the package root. Keep that local file out of Git.

The JSON structure is:

Fill the fields required by the selected task instances. In Docker mode, the runtime input is mounted read-only under /run/payskills/inputs/ and exposed through the environment variable declared by the task instance, such as ALIPAY_SANDBOX_KEYS_FILE. When a task declares workspace_path, the runtime also copies the file into /workspace before the agent starts.

Docker

enabled: run task instances in Docker; the release default is true; build: build each selected task instance's task/Dockerfile; image: existing local image used when build is false; network: optional Docker network name

With Docker enabled, the runtime mounts /task_instance, /workspace, /output, and /opt/payskills_runtime inside the sandbox. Task images must include bash, python3, and git.

Selecting Task Instances

Specify task instance IDs in config/config.yaml:

Run ./run.sh --dry-run to confirm the final selection and helper requirements.

📦 Run Outputs

Each run creates:

summary.json: run index and aggregate task status; result.json: score and per-rubric pass/score results written by the evaluation; execution.json: runtime status for the run, diff, and evaluation phases; logs/: raw phase logs; agent_trace.json: standardized turn-level agent trace; agent_events.jsonl: detailed agent event stream; agent_evidence.json: compact evidence used by LLM-assisted evaluation; patch.diff, changed_files.txt, code_files/: workspace changes produced by the agent

Run outputs may contain complete agent inputs, outputs, commands, and runtime sandbox material. Treat the entire run directory as sensitive data and do not publish it without review.

🧩 Contracts

Task Instance Contract

Each task instance follows this layout:

task/run.sh invokes the selected agent and writes non-scoring artifacts. evaluation/evaluate.sh runs deterministic and LLM-assisted evaluations and writes $OUTPUT_DIR/result.json.

result.json is the scoring contract. It contains numeric score and max_score, a summary, and a rubrics list. Each rubric records its own pass state, score, maximum score, type, and diagnostic message.

When a task instance calls payskills-agent drive-turns, the agent helper writes agent_trace.json, agent_evidence.json, and agent_events.jsonl under artifacts/. Use agent_evidence.json for LLM-assisted evaluation, agent_trace.json for turn-level summaries, and agent_events.jsonl for detailed debugging.

Legacy task layouts such as environment/Dockerfile, a task-root run.sh, or a task-root test.sh are rejected.

Package Contract

manifest.json is the machine-readable package contract. It records:

top-level runner commands; required task instance paths; configuration and helper CLI contracts; standard run artifact paths

./run.sh --doctor validates the manifest, package layout, selected tasks, configuration, required commands, credentials, runtime inputs, and Docker readiness before creating a run.

🛠️ Troubleshooting

Docker daemon is not reachable

Start Docker and confirm that docker info succeeds as the current user.

Agent or judge model/API key is missing

Follow the ERROR and HINT lines from ./run.sh --doctor. Check agent, judge, and the variables referenced in config/.env.

Alipay sandbox input is missing

Configure runtime_inputs.alipay_sandbox and fill the referenced environment variables or local JSON file required by the selected tasks.

Validate without starting an evaluation

PIBench is an open-source benchmark for evaluating AI coding agents on realistic, end-to-end payment integration tasks.