MiniMax M3.1 Flash Faces Memorization Doubts After Low-Price Leaderboard Entry

By | Shuzhi Benliu, Author | Shang Ye, Editor | Shen Xiao

"You know all the old questions, but you're clueless on new ones?"

After MiniMax's new model M3.1Flash was released on October 19, discussion around it quickly shifted from high leaderboard scores to suspicions of memorizing questions.

In a post, a user relayed test feedback from AI evaluation blogger "Boss Cat": "How to put it, m31f gets near-perfect scores on old questions and near-zero on new ones."

In related discussions on the Linux.do forum, some debated whether the model had memorized questions, while others went further, speculating about whether the vendor had identified evaluation questions from the input and applied special handling for the tests.

The model had just delivered a report card showing clear improvement. On the day of release, independent evaluation firm Artificial Analysis (hereafter AA) gave M3.1 Flash a composite intelligence index score of 42, 13 points higher than the M3 released in June and only 3 points lower than DeepSeek V4.1 Flash-1017.

But a modified-question test demonstrated by user Binkg further deepened the suspicion. He selected a production management question from AA's public sample questions and changed the conditions that determined the answer. DeepSeek V4.1 Flash-1017 and Gemini 4 Argon changed their answers accordingly, but M3.1 Flash still gave the original question's answer.

Beyond the doubts, the new model also saw extensive trial use. According to reports, M3.1 Flash processed more than 100 trillion tokens within 72 hours of launch, during which multiple platforms offered limited-time free access.

As of October 25, on OpenRouter's seven-day leaderboard alone, its free version had processed about 13.7 trillion tokens, ranking third; DeepSeek V4.1 Flash-1017 ranked first with 33.6 trillion tokens.

On one side was massive usage, on the other the "memorizing questions" doubts. We also tried the modified question ourselves: when asked in English, MiniMax answered incorrectly; when switched to Chinese in a separate conversation, it answered correctly.

Before the doubts arose, M3.1Flash had already attracted a wave of attention under another name.

In September, the anonymous model Space Bunny Alpha, known to everyone as "Space Rabbit," landed on OpenRouter with free access. Space Rabbit then ranked first on OpenRouter in usage for ten consecutive days, running up 56.7 trillion tokens and setting the platform's record for an anonymous model.

On October 19, MiniMax's official account posted "Welcomeback.SpaceBunny." to reveal the answer: Space Rabbit was an early version of M3.1Flash, which had already processed more than 200 trillion tokens during its anonymous public beta. The company further disclosed that the official version continued to optimize capabilities such as coding and multi-step agent tasks, while cutting costs by 70%.

In AA's tests, running the same task cost about $0.22 on average for M3.1Flash, less than half of M3's $0.51 and even cheaper than DeepSeek V4.1 Flash-1017's $0.26.

At the official API standard price, M3.1 Flash's input and output prices per million tokens are $0.15 and $0.60 respectively, more than half cheaper than M3. Compared with DeepSeek V4.1 Flash-1017's peak prices, M3.1 Flash's input and output prices are also half as low.

As a result, some users joked with "Liang Wengu in idle times, Yan Junfeng in busy times," teasing the two Flash models' close-quarters price war.

But feedback from the well-known blogger Boss Cat quickly shifted the ignited sentiment from "how many points did it improve" to "how did those scores come about."

The leaderboard focuses on models' logic, math and coding abilities, using a private question bank iterated monthly to avoid publicly available original questions online. The question types lean toward real-world scenarios, such as tracing clues from server logs, map pathfinding with constraints, and inferring game rules on one's own in a match with no instructions.

Its evaluation rules are that each question is tested three times, with the best result taken; not only must the answer be correct, the derivation process must also be compliant, and simply guessing the right answer does not count. The continuously rolling question bank retains some old questions while incorporating new ones, used to compare and observe changes in model capabilities.

His core doubt this time was the split in M3.1Flash's abilities on old versus new questions.

He observed two anomalous phenomena:

- The model scored near-perfectly on retained old questions, while scoring near zero on newly added questions;

- At the same time, the token consumption pattern was also abnormal: the model consumed more than 40,000 tokens per question on average, yet for some of the difficult questions, it gave the correct result using only about 10,000 tokens.

Normally, the longer the reasoning steps, the higher the output tokens consumed. By his observation, difficult questions were answered correctly using only about a quarter of the average usage, while new questions failed on a large scale. This made users wonder: is the model actually re-solving the problems, or is it recognizing old questions it has seen before?

Another AI evaluation blogger, Binkg, further tested the possibility of question memorization with a modified-question experiment.

He selected a public evaluation question from AA-Omniscience. The original question asked: in the capacity planning system of production management, which long-term planning method predicts resource requirements such as the number, location, and scale of new factories? The answer is Resource Requirements Planning, abbreviated RRP.

Binkg replaced "long-term" with "medium-term," and replaced the number, location, and scale of new factories with the manpower, equipment, and capacity needed to complete the master production schedule. The answer to the modified question should be Rough-Cut Capacity Planning, abbreviated RCCP.

In the results Binkg posted, DeepSeekV4.1Flash-1017 and Gemini4Argon both answered RCCP in line with the new question, and only M3.1Flash still answered RRP.

(Image from Binkg's post)

What also made Binkg suspicious was the jump in the knowledge-question score.

In AA-Omniscience, the previous-generation model M3 had an accuracy of 16.7%, while M3.1Flash reached 56.2%, an increase of 39.5 percentage points at once, with more than three times the number of correctly answered questions.

As a reference from the same knowledge test recently, KimiK3.1Pro, released on October 23, rose from 47.6% for KimiK3 to 51.0%, an increase of 3.4 percentage points.

Binkg also raised a guess about the source of the questions.

After the previous-generation model M3 was released in June, AA evaluated models through an online interface. If evaluation requests go through an interface operated by the vendor, the vendor technically has an opportunity to access the questions. He suspected that MiniMax may have recognized these evaluation requests and used the questions for subsequent training.

Beyond the question-memorization controversy, AA's test also left a set of seemingly contradictory numbers.

AA-Omniscience mainly tests factual knowledge, and also tests how models answer when they encounter knowledge gaps. In an exam that generally only calculates accuracy, wrong answers and unanswered questions score the same, and guessing at an answer at least has a chance of scoring.

AA-Omniscience changed this rule: correct answers add points, wrong answers deduct points, and partially correct or unanswered questions neither add nor deduct. Before the test begins, it also explicitly reminds the model to answer only when confident, and that giving up is better than answering incorrectly.

The test results show that M3.1Flash had an accuracy of 56.2%, yet its hallucination rate was as high as 92.8%. Roughly converting using rounded numbers, out of every 100 questions, M3.1 Flash answered about 56 correctly and 41 incorrectly, with the remaining about 3 partially correct or abandoned. The 93% refers to the proportion of errors among the latter 44 questions.

Compared with several models with similar overall scores, M3.1 Flash answered the most knowledge questions correctly and also had the highest hallucination rate. For example, its accuracy differs from Gemini 3.8 Flash by only 1.6 percentage points, yet its hallucination rate differs by 37.6 percentage points.

The previous-generation model M3 often chose not to answer in this test, and its hallucination rate was only 18.4%. Converted per 100 questions, about 17 were answered correctly, 15 incorrectly, and the remaining about 68 partially correct or unanswered. For M3.1 Flash, correct answers rose from about 17 to 56, and incorrect answers also rose from about 15 to 41.

However, in the data analysis evaluation jointly built by Plotly and Mila researchers, minimax M3.1 Flash scored high: it answered 41 of all 43 questions correctly, tying with models such as Claude Opus 5.5 and Gemini 4 Argon.

Previously, its anonymous early version "Space Rabbit" passed 38 questions, on par with models such as Zhipu GLM 5.3 Flash and Alibaba Qwen 3.8 27B, making it one of the best-performing open-weight models in this test.

This set of questions simulates real business scenarios, giving the model access to a simulated wind power data warehouse with 20 million records and requiring it to independently query, calculate, and answer business questions. At the same time, traps are embedded in the questions, posing questions that the dataset cannot support, such as asking a model that only has wind power data to calculate solar power generation, in order to test whether the model can discover missing data, identify false premises, and restrain hallucinated output.

The testers also embedded five patterns in the data that needed to be actively discovered. Space Rabbit in September found only two, while M3.1 Flash found all of them, but the median time per question increased from 36 seconds for Space Rabbit to 3 minutes.

However, the "question memorization" accusation still lacks support from request logs or training records. A model answering with an old answer may involve questions in the training materials, or it may be conceptual confusion.

To this end, we also replicated Binkg's modified-question experiment.

We asked MiniMaxM3.1Flash, Doubao Fast Mode, and DeepSeekV4.1Flash-1017 the modified original English question separately.

As with Binkg's test results, Doubao and DeepSeek both changed their answers in line with the new question, while MiniMax still answered RRP.

(Binkg's modified new question, screenshots of the three vendors' actual tests in English)

Next, we translated the question into Chinese, kept the question unchanged, and opened a new conversation with each again. This time, Doubao, DeepSeekV4.1Flash-1017, and MiniMaxM3.1Flash all answered correctly.

(Binkg's modified new question, screenshots of the three vendors' actual tests in Chinese)

In addition to question answering, we also assigned MiniMax and DeepSeek a creative task: to make a GIF animation of about 4 seconds.

Both products received the same two reference images and the same set of requirements: on a grassy field with thin mist, an orange cat wearing a pink bow, silver armor, and a white cape stands holding a sword, while a giant dragon lowers its head and approaches, opening its mouth and roaring. Preserve the reference appearance, highlight the size contrast, keep the movements natural and coherent, and directly deliver a playable GIF animation.

(Close-up of the cat design and scene reference images)

DeepSeek V4.1 Flash-1017 delivered a GIF of about 6.2 seconds. The reference appearance was consistent, but it mainly switched between two compositions, with relatively little character movement.

M3.1 Flash delivered a GIF of about 4 seconds, with more character movement, but the designs of the cat and dragon differed from the reference images.

We asked MiniMax to redo it according to the reference images, again emphasizing the height difference between the cat and the giant dragon. MiniMax then listed the modifications, saying that the pink bow, star-patterned silver armor, and white cape had all been preserved, and also delivered horizontal and vertical versions.

In the vertical version, the giant dragon approaches from above the head, and the kitten stands at the bottom of the frame, making the size contrast more obvious. But the cat's face and clothing still did not reproduce the reference images.

Finally, we explicitly invoked MiniMax's H3 video generation tool, uploaded the original video, and required it to replicate the characters, camera, and music of the first 5 seconds. This time, the character appearance and music output by MiniMax matched and replicated the reference video.

On October 23, MiniMax opened the model weights of M3.1 Flash according to the plan previously announced, offering a follow-up solution that anyone can verify. Researchers with deployment conditions can retest on their own devices, without the questions needing to go through MiniMax's online interface.

This adds conditions for verification, but the public weights did not simultaneously make all training materials public. Why that modified question was answered incorrectly, and why it was answered correctly when switched to Chinese, still requires more testing to explain.