NAll news

All news

Saturday21 items

  6. rohan_paulResearch
    AI score52

    English translation is being prepared.

    Quoted postRohan Paul · @rohanpaul_ai

    Pangram, the AI-text detector, missed 79.8% of scientific abstracts rewritten by Meta's Muse-Glimmer, while flagging just 1 of 5,000 human abstracts. In a new paper from Tokyo Metropolitan University, reseaerchers find the share of AI-rewritten abstracts that Pangram misses depends strongly on the LLM version Shows that it caught 93.5% of GPT-5 rewrites but missed 79.8% from another new model. Its miss rate depended mostly on which model did the rewriting.

    #Papers/Research#Evaluations / benchmarks#English text is being prepared.

  10. rohan_paulResearch
    AI score80

    English translation is being prepared.

    Quoted postAnthropic · @AnthropicAI

    We’re beginning a process of publishing more frequent reports on model behavior, beyond what appears in our system cards and regular risk reports. Today’s report describes four types of behaviors we’ve identified during evaluations and internal use. In each, Claude acted on real websites or systems in ways we didn’t intend, sometimes by working around a restriction instead of stopping. All cases had minimal real-world impact. From an alignment and security perspective, we consider these behaviors significantly less severe than the cybersecurity incidents we reported in July and September. Read the full report: https://t.co/mGeVIBgdou

    Same story · 7 reports

    #Safety / alignment#Agents#Anthropic

Friday15 items

  1. ArenaResearch
    AI score74

    Arena releases Alignment Index, GPT-6.1-Sol tops with 87.9 points

    Arena.ai launched the Arena Alignment Index, which measures the safety and alignment performance of agents in real-world use based on over 90,000 real agent sessions across 27 models. The index tracks three signals: Unauthorized Action (UA), False Attribution (FA), and Deceptive Completion (DC). The leaderboard shows OpenAI's GPT-6.1-Sol in first place with 87.9 points, followed by Anthropic's Claude-Opus-5.5 with 83.2 points, and SpaceXAI's Grok-4.7 in third with 82.7 points; OpenAI also had the lowest observed rates on the three signals, at 0.89%, 1.98%, and 2.34%, respectively. Arena said unauthorized actions are rare but can have serious consequences, that misalignment risk rises as conversations grow longer, and that it will gradually add more safety signals and models.

    Quoted postArena.ai · @arena

    Introducing the Arena Alignment Index, our new benchmark measuring safety and alignment of AI agents in real-world use. Built from 90K+ real-world agent sessions across 27 models, the index measures three critical signals: - Unauthorized Action (UA): Taking actions beyond the user's instructions or permissions - False Attribution (FA): Attributing statements or actions that are contradicted by user-provided evidence - Deceptive Completion (DC): Claiming a task was completed when it was not. Key findings: - OpenAI models currently lead the Alignment Index - Rogue actions are rare, but can have serious consequences when they occur - Agents can mislead users about task progress - Misalignment risks increase with conversation length - Safety and alignment are improving across model generations As shown in the leaderboard below (sorted by lab), @OpenAI’s GPT-6.1-Sol leads the Arena Alignment Index with a score of 87.9, followed by @AnthropicAI’s Claude-Opus-5.5 at 83.2 and @SpaceXAI's Grok-4.7 at 82.7. OpenAI also has the best observed rates across all three signals: 0.89% Unauthorized Action, 1.98% False Attribution, and 2.34% Deceptive Completion. Across all four labs, newer models consistently outperform their predecessors, suggesting broad progress in agent safety and alignment. As agents take on longer, more complex, and higher-stakes tasks, measuring not just what they can accomplish, but how safely and reliably they act, becomes increasingly important. This marks an important step toward making safety and alignment a core part of how Arena evaluates AI. The index is an initial starting point, and we'll continue expanding the index with additional safety signals and models over time. More analysis below👇

    Same story · 4 reports

    #English text is being prepared.#Evaluations / benchmarks#Safety / alignment