AI News

Trending story

Sakana AI Releases Japanese Financial Benchmark JFinBench

1 reports1 story updatesUpdated Nov 16 17:13

Understand this story

AI overview

Sakana AI built and released JFinBench, a Japanese financial benchmark based on the 2026 editions of six Japanese financial qualification exams, comprising 321 questions in total, including 193 multiple-choice questions, 128 written-answer/essay questions, and 12 questions containing images. Sakana AI evaluated 12 LLMs: Claude Fable 5 ranked first with 95.8 points, GPT-6 Astra scored 93.6, and the best open-weight model, Kimi K3.1 Pro, scored only 83.4, about 10 points lower than GPT-6 Astra; the main reason for lost points was insufficient knowledge of Japanese institutions rather than calculation. The construction process was carried out by coding agents, which handled PDF conversion, with humans verifying every question; the six exams were completed in about 12 hours.

AI-generated from public reports

Latest development · Report

Sakana AI Releases Japanese Financial Benchmark JFinBench: Kimi K3.1 Pro Trails GPT-6…

Developments

1 confirmed developments

  1. Sakana AI releases Japanese financial benchmark JFinBench

    Source:Sakana AI:Blog

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 16

  1. Sakana AI:BlogFirst-party

    Sakana AI Releases Japanese Financial Benchmark JFinBench: Kimi K3.1 Pro Trails GPT-6…

    Sakana AI built JFinBench, a Japanese financial benchmark based on the 2026 editions of six Japanese financial qualification exams, comprising 321 questions in total, including 193 multiple-choice questions, 128 written-answer/essay questions, and 12 questions containing images. After evaluating 12 LLMs, Claude Fable 5 ranked first with 95.8 points, GPT-6 Astra scored 93.6, and the best open-weight model, Kimi K3.1 Pro, scored only 83.4, about 10 points lower than GPT-6 Astra; its main source of lost points was insufficient knowledge of Japanese institutions rather than calculation. The construction process was carried out by coding agents for PDF conversion, with humans verifying all questions; the six exams were completed in about 12 hours.

Related stories

  1. Related storiesQwen Releases Qwen4-Max Preview Version
  2. Related storiesGoogle launches Gemini 4 Hydrogen on Antigravity
  3. Related storiesBaidu Wenxin releases ERNIE 5.5 Flash Preview
  4. Related storiesThinking Machines Lab releases Inkling 2 Small