Google DeepMind and Stanford paper finds that colored shapes in prompts can sway the code models write
A new paper from Google DeepMind and Stanford finds that adding task-irrelevant solid-colored shapes to prompts changes the coding choices of vision-language models. The study tested 288 shapes; on a binary tree problem, a red heart raised the proportion of recursive solutions from 0% to 100% for Claude Opus 4.7 and GPT-5.4; when the shapes did influence Gemini 3.1 Pro, 71.5% of its thinking traces still described the image as irrelevant. The authors recommend running with-image versus without-image control tests on agents that read images, and not expecting reasoning traces to signal visual influence.
Why read this
AI score 64
The original text provides specific experimental numbers showing that irrelevant images can change a model's coding choices, and that the model's reasoning traces may not signal this influence.
Topics
Body
so interesting. 🤔
New Google DeepMind and Stanford paper finds that a red heart picture in the prompt can decide how AI write code, that simple shapes in images can steer AI on unrelated tasks, often without the models saying so.
plain colored shapes in a prompt change how AI models write code, though the shapes have nothing to do with it.
Vision-language models now read screenshots and images as part of normal work. If an unrelated picture changes the model's choice, builders need to see it, ideally in the model's own reasoning.
They added a plain colored shape, like a red heart, to a coding prompt and told the model to look at it. Across 288 shapes, they tracked how often Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.4 chose recursion.
A red heart moved Claude Opus 4.7 and GPT-5.4 from 0% recursive solutions to 100% on a binary tree problem. When a shape did steer Gemini 3.1 Pro, 71.5% of its thinking traces called the image irrelevant.
If your agent sees images, test its decisions with and without them, and do not count on reasoning traces to flag visual influence.