Understand this story
QM-LibBench proposes a benchmark for cross-chip library migration, covering 85 real operators in OpenCV, ncnn, and libjpeg, designed to evaluate the library migration capabilities of AI agents. The benchmark comes from an ICML 2026 paper, referred to as CLAM-Bench during review. Evaluations show that 12 "agent + large model" combinations on average got only 20.88% of tasks right on the first attempt, with the best combination at 37.65%; after adding the requirement of no slowdown, nearly all dropped close to 0. The paper attributes the root cause to agents merely imitating the surface characters of source code rather than understanding the semantics of vector instructions, and proposes a multi-agent framework FSCM that first clarifies the library dependency structure before generating code to improve accuracy.
AI-generated from public reports
Developments
1 confirmed developments
QM-LibBench proposes a cross-chip library migration benchmark
Source:English text is being prepared.
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 22
Alibaba Cloud Developer Community AI
ICML 2026 | QiMeng-LibBench: 85 real operators show how far AI still is from migrating libraries across chips
The ICML 2026 paper proposes QiMeng-LibBench (referred to as CLAM-Bench during review), which uses 85 real operators from OpenCV, ncnn, and libjpeg to evaluate the cross-chip library migration capability of AI agents. The 12 "agent + large model" combinations on average got only 20.88% of tasks right on the first attempt, with the best combination at 37.65%; after adding the requirement of no slowdown, almost all of them dropped close to 0. The paper points out that the root cause is that agents only imitate the surface characters of source code rather than understanding the semantics of vector instructions, and proposes a multi-agent framework FSCM that first clarifies the library dependency structure before generating code to improve correctness.