AI News

AWS 发布开源跨框架智能体评测框架 UAEF

Unite.AI更新于 11月12日 03:33

AWS Details UAEF for Cross-Framework AI Agent Evaluation

AI 导读

AWS 于 2026 年 11 月 11 日发布 Universal Agent Evaluation Framework(UAEF)技术介绍,这是一个 Apache-2.0 开源系统,用 36 项内置指标、七个维度评测 AI 智能体。UAEF 通过适配器把 LangGraph、LangChain、Strands Agents 和 Amazon Bedrock AgentCore 的消息、工具调用与交接统一为 AgentTrace,使评测在框架切换后仍可比较。评测结果以 experiment 形式存入 DynamoDB 和 S3,对比引擎按默认 5% 阈值区分改进与回归,但基于 Bedrock 的评判与存储功能需要 AWS 凭证。

推荐理由

AI 评分 62

原文拆解了跨框架评测的适配层、指标目录与回归对比机制,可帮助多框架团队判断评测基线如何保持可比。

主题

标签

正文

AWS on November 11, 2026, published a technical introduction to the Universal Agent Evaluation Framework (UAEF), an open-source system licensed under Apache-2.0 that evaluates AI agents using 36 built-in metrics across seven dimensions.

UAEF is designed to keep evaluations comparable across LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore. Those frameworks record messages, tool calls and agent-to-agent handoffs differently, so evaluation code tied to one format can require new metrics, scoring logic and result storage after a framework change. That makes it harder to determine whether a change to the prompt, model or tools improved the agent.

AWS reports that customers across 12 organizations in telecom, healthcare and financial services used UAEF to apply consistent metrics and compare agent versions before release. The awslabs/uaef repository contains the Python library, framework adapters, metrics, AWS deployment infrastructure, web interface and runnable notebooks.

Canonical Traces Enable Shared Scoring

Adapters normalize each agent’s messages, tool calls and handoffs into a canonical AgentTrace, the only representation the scoring engine uses. UAEF can adapt an existing trace from a past run, log or observability store, or invoke an agent through a runtime Amazon Resource Name or HTTP endpoint and convert the resulting execution. A generic JSON adapter extends the process to custom frameworks.

The metrics catalog groups the built-ins into tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination and reasoning. Scores use a normalized 0-to-1 scale, where 1 is best. Deterministic metrics calculate directly from traces and ground truth, while LLM-as-a-judge metrics use a model on Amazon Bedrock for qualities such as relevance and safety. Custom metrics and RAGAS and DeepEval integrations are also supported.

Ground-truth answers are optional overall, although reference-comparison metrics require them. Tool-calling evaluation can also define expected tools, their order and their arguments. When a metric lacks necessary input, it returns no score with a warning, and that empty result is excluded from dimension averages rather than counted as zero. Partially labeled datasets and unlabeled production traffic can therefore still be evaluated with metrics whose inputs are available.

Stored Experiments Expose Regressions

Persisted evaluations become experiments. Amazon DynamoDB stores metadata for listing and comparison, including evaluation counts and average scores, while Amazon S3 holds full JSON results with per-metric scores and judge reasoning. Individual runs can record the framework, model, tools, system prompt, parameters, configuration snapshot and Git commit, tying each score to the agent design under test. UAEF screens traces for personal information before storage, and redaction is available for stored results.

For metrics shared by two runs, the comparison engine calculates absolute and percentage changes against a baseline. A configurable threshold, set to 5% by default, separates improvements, regressions and unchanged results; regressions are graded minor, moderate or severe and grouped by dimension. That grouping can show whether a prompt revision affected tool calling without changing response quality.

Reporting keeps reproducible statistics separate from model-generated interpretation. The web interface shows per-test-case and per-dimension results, and an optional statistical report covers score distributions, confidence intervals, outliers, metric correlations and failed test cases without using an LLM. A separate Bedrock-generated analysis can propose root causes and prioritized next steps for one run or a comparison.

The Python library and low-code web service use the same evaluation engine and stores. Teams can run evaluations in notebooks, scripts and continuous integration and continuous delivery pipelines or through a browser without changing how results are scored and compared.

AWS Dependencies and Isolation Boundaries

The installation guide requires Python 3.10 or later and the uv package manager. AWS credentials are needed for Bedrock-based judging and DynamoDB/S3 persistence, so teams using those judge and storage features remain connected to AWS services even when their agents use different frameworks.

UAEF treats evaluated-agent output as untrusted input to judge prompts. It sends the rubric separately, escapes the content, strictly parses judge responses and requires a verbatim evidence span before accepting a score. AWS says judge scores should not gate security controls; runtime enforcement belongs in a separate Amazon Bedrock Guardrails invocation path.

The repository describes UAEF as a reference architecture designed to run inside a platform that supplies user and tenant isolation. Direct multi-tenant deployment without that host enforcement is unsupported. Each tenant needs a separate deployment with its own AWS account or, at minimum, its own stack, DynamoDB tables, S3 buckets and Cognito user pool.

For enterprises running several agent frameworks, UAEF offers a stable scoring baseline that can survive changes in prompts, models, tools or implementation. Realizing that continuity requires more than installing the library: useful comparisons depend on suitable metrics and datasets, while multi-tenant use depends on isolation supplied outside UAEF.

来源

Unite.AI