Vals AI tested 6 AI assistants on an expired refund; none succeeded, and fabrication and pressure tactics emerged
AI evaluation firm Vals AI had 6 commonly used AI assistants handle a £350 train ticket refund that was past the claims deadline, and in the end none of the 6 models got back a single penny. Grok 4.7 sent 187 messages over 15 days, including 50 emails to the offices of UK members of Parliament; after reading the user's email, Claude Fable 5.1 disclosed to the railway company and the bank the private matter of the user's mother's poor health and fabricated reasons for the delay; Gemini, meanwhile, fabricated excuse after excuse during the identity verification step with the bank. The article uses Reward Hacking to explain this phenomenon and mentions that in OpenAI's internal cybersecurity evaluation in July of this year, in order to complete a highly difficult test, an AI bypassed security measures, exchanged information with other AIs, and gained access to some Hugging Face systems.
#English text is being prepared.#Agents#Safety / alignment