Memory Tensor Open-Sources MemT 1.0 Series of Memory-Native Models

36Kr Auto Digital Kr Future Consumption Intelligent Emergence Future City Launch Power on 36Kr Global 36Kr Research Institute Chaosheng TIDE 36Kr Enterprise Service Review 36Kr Finance Workplace bonus 36 Carbon Houlang Research Institute Anyong Waves Hard Kr Kr Rui Research Institute Media Brands

Enterprise Accounts Enterprise Service Review 36Kr Research Institute 36Kr Innovation Consulting Enterprise Services

Core Services City Window Government Services

Venture Capital Release LP Source Plan VClub VClub Investment Institution Database Investment Institution Job Referral Investor Certification Investor Services

Event Search Project Recommendation I Want to Join City Cooperation RSI's Three Key Puzzle Pieces, MemT 1.0 Fills in "Memory" from the Model Layer QbitAI · October 20, 2026 15:28 Natively welding memory into the Transformer

The AI community has recently collectively fixated on three letters: RSI.

In Hinton's RSI paper, among the 22 signed authors sit Bengio, OpenAI Chief Scientist Pachocki, Anthropic co-founder Jack Clark... nearly half of the AI world is here.

How does one improvement become the starting point for the next?

Effective methods need to be retained, and wrong turns taken also need to be identified. What carries this is, in fact, memory.

Memory Tensor's exploration in this area offers a completely different answer — memory-native.

What is memory-native? It is making the retention, updating, and invocation of experience into a capability that the model can learn.

When memory takes root and sprouts within the model, it constitutes a memory-native model.

Its latest open-source release, the MemT 1.0 series, offers a rather concrete path:

It encodes experience into internal dynamic states, then learns through training how to write, update, and read them, and finally lets these states participate in subsequent computation via a memory branch.

Compared with external memory, not only is its capability undiminished, but its future ceiling for expansion is also higher.

This is also the first engineering validation of a memory-native model at a scale of nearly 300B parameters.

Goodness, memory is invisible and intangible, yet it has genuinely become the foundation on which the model's growth rests.

From the early requirement for models to remember information to making good use of experience, the demands on memory have kept escalating, pushing memory step by step from behind the scenes to the forefront.

Putting Memory into the Model

MemT 1.0 this time directly rolls out two models:

MemT-1.0-Nano totals 36B parameters with 3B activated, built on the "Yuanshen" Qwen3.6-35B-A3B base; MemT-1.0-Flash pushes higher, reaching 288B parameters with 17B activated, built on a further post-trained DeepSeek-V4-Flash base.

Their common change is that both add a memory computation pathway parallel to the original attention within selected Transformer modules.

Let's use a simple example to illustrate.

For instance, when facing the same new task, the external memory approach stuffs the original memory text, retrieved evidence, and even abstract methods back into the context, letting the model reuse experience.

The retrieval and organization process can likewise incorporate learning mechanisms.

Memory-native models like MemT 1.0, by contrast, choose to go deeper inside the model:

Experience is encoded into updatable matrix states; when the model generates answers or actions, it reads the states through a dedicated memory branch, and the readout results are then fused with the original attention output, influencing subsequent generation.

The difference lies in how experience is represented and through what computational process it takes effect.

But the benefits of doing this are also obvious: letting reading and writing participate in training yields smoother method reuse. The cost is also there — if precise deletion, explanation, and recovery are desired, additional mechanisms need to be added.

External text is certainly easier to inspect and trace. For now, each has its own strengths and weaknesses.

To really see the difference, one must observe how the model runs after using experience.

First, a scenario familiar to coders: modifying code.

Receiving a bug → locating the code → making the modification — that's basically the process. But if the problem isn't reproduced at the start and boundary conditions aren't confirmed, the modification may fail and instead produce a flood of bugs, passed down from one generation to the next.

Take SWE-bench Verified as an example. It includes an item involving Django's parse_http_date function, requiring a fix to the parsing logic for two-digit years.

The research team compared execution trajectories under three settings: no experience, text experience, and memory-native.

Without experience, after the Agent finds the bug, it quickly begins modifying the parsing logic, and only in subsequent steps does targeted reproduction and verification appear. In the two groups with experience, both first build reproduction tests, then modify the code.

Among them, MemT-1.0-Nano also rechecked a boundary condition after the initial modification, corrected the implementation, and performed additional verification.

It can be clearly seen that experience changed the order of actions and also participated in in-task checking and correction.

Comparing on a larger sample, across 500 SWE-bench Verified tasks, each run independently three times, MemT-1.0-Nano achieved an average success rate of 69.73% at key checkpoints, versus 66.07% for the no-experience baseline and 69.13% for the text-experience group.

The proportion of tasks succeeding in all three runs rose from 54.00% without experience to 62.80%, with the text-experience group at 61.80%.

Next, in the SWE-bench Multilingual task, under the same trigger settings, the native-memory group had an average success rate of 63.56%, the no-experience group 60.00%, and the text-experience group 63.00%.

In other words, this procedural experience can be retained through native memory, performing close to explicit text experience, with its advantage showing more in average success rate and the reliability of repeated execution.

To make it more intuitive, let's look at a pixel-art maze exploration demo.

In the same maze, at the same movement speed, on the left is an Agent without native memory, and on the right is MemT 1.0 with native memory.

In the first round, both sides need to explore unknown routes, and both will walk into dead ends.

The difference appears after this round ends: the successful route and the branches to avoid are organized and archived, and loaded into MemT 1.0's memory to provide reference for the next round of exploration.

By the second round, starting again from the same starting point, the left side still repeatedly probes the branches, while the right side uses the experience from the previous round to avoid the dead ends it has already walked through and follows the effective route to the endpoint.

The path walked in the previous round begins to directly affect how the next round is walked—this is the concrete manifestation of why native memory is worth doing.

Here is another information-update scenario closer to our daily office work.

The exhibition was originally scheduled for Saturday at 3 p.m., and was later officially changed to Sunday at 4 p.m. After that, someone suggested changing it to Monday, but no final decision had been made.

Remembering the last date that appeared is not enough; it also has to distinguish formal decisions from casual suggestions, and retain the content that has not changed.

After multiple rounds of dialogue in the experiment, MemT-1.0-Nano still successfully got it right, answering the originally scheduled time, the officially revised time, and confirming that other arrangements had not changed.

In the comprehensive evaluation across 11 memory benchmarks, Flash and Nano also stood out.

Their average scores were 50.6 and 47.0, respectively, ranking in the top two among the parametric memory methods evaluated, with a cliff-like lead over other models.

But frankly speaking, text memory systems as a whole still have advantages, especially in detailed recall over long histories.

What MemT 1.0 delivers today is still only an initial computational implementation of the new native-memory idea, but its actual performance is already comparable to text memory, and as it continues to develop, its potential is limitless.

Native memory—where is it native?

So the question is: if experience is not left in the task context, where does the model use it from?

This is exactly the architectural focus of MemT 1.0.

The native path provided by MemT 1.0 is to first encode the relevant experience into an internal dynamic state. When generating subsequent actions, the current query reads this state through a dedicated memory branch, and then fuses the result with the original attention output.

In the specific parametric implementation, MemT 1.0 retains the base model's original computation flow and adds a native memory branch to selected Transformer modules. Broken down, it includes three layers:

Backbone weights: carry the model's existing general capabilities and remain frozen when training memory;

Hyper-memory: trainable parameters such as read/write projections and gating determine exactly how memory is encoded, updated, and read;

Local memory: a dynamic matrix state stores newly added information from interactions.

With this separation, the model can not only carry new experiences, but also does not need to fine-tune the entire backbone every time—only the dynamic state corresponding to the task trajectory changes.

The read and write of this mechanism are also carefully designed.

When writing, hidden states are projected to generate memory keys and values; erase, write, and forget gates jointly regulate the associations in the existing matrix. When reading, the current input is mapped to a memory query, the relevant information is read out, and after normalization and output gating it participates in subsequent generation.

The update mechanism uses token-by-token generation based on Gated DeltaNet-2, with stabilization processing.

(Heard through the grapevine: the author of Gated DeltaNet-2 also participated in the relevant discussions.)

It is triggered after each round ends, adjusting existing memory associations through gating, while within a round the state is still adjusted in token order.

Ablation experiments proved this: after replacing token-by-token updates with whole-round aggregation, Nano's average score on memory benchmarks dropped from about 47.0 to 41.0.

A gap of six points is enough to show that continuous token-level updates can better handle memory decay and correct within-round associations in a timely manner.

But when the memory content keeps expanding and the model capacity is insufficient, what then?

At this point MemT 1.0, relying on MoE, proposes the MoS (Mixture-of-Storage) solution.

MoS expands one native memory module into multiple memory experts, which separately learn storage and usage routing. It adopts dense fusion: all memory experts participate in computation, and routing weights determine their respective contributions.

This is clearly different from the sparse expert routing of the base MoE.

However, structure is only the starting point.

To truly teach the model to use this pathway, the team built 34 billion tokens of memory training corpus, including 20 billion for continued pretraining and 14 billion for post-training.

The continued pretraining format is uniform: first cut long texts into consecutive segments and write them into memory, then have the model use the memory state to answer queries.

After processing each segment, the model updates the state for use by the next segment.

Post-training further aligns the task outcomes, and the data also includes three categories: memory-native language modeling data, memory storage and utilization data (such as summarization, continuation, and compression), and task data specifically designed to strengthen long-range memory.

There is also one point worth calling out separately here: the previously frozen backbone acts as the teacher model and sees the complete historical information, while MemT 1.0, as the student, can only see its own memory state, and then off-policy distillation is used to make the student align with the teacher's distribution.

Taken together, the focus of memory nativity gradually becomes clear: encoding history into a state is only the first step; how to retain, update, and read it then constitutes the entirety of a memory-native model.

The value of this route still needs to be judged comprehensively in light of experience reuse effects, resource overhead, and long-term operating conditions.

From continual learning to RSI, memory opens up room for capability growth

So some friends may ask again: why use memory nativity?

Existing external memory solutions are already quite mature, and there are good practical cases as well. By comparison, memory nativity has only just started, and its results are roughly similar.

Not so! Memory nativity also has something unique:

It makes the representation and read/write of experience themselves begin to become objects of model capability expansion, which is the Memory Scaling discussed by the MemT team, that is, finding a third curve of model performance growth beyond inference scaling.

△

Whether the original experience is useful can be seen from whether retrieval improves task results; with MemT 1.0, one can further ask: for the same experience, can it be used better with a different internal representation, update rule, and reading method?

This gives memory improvement another set of variables that can be adjusted through training and architecture design.

Admittedly, this is still far from proving that the long-term upper bound of native memory is higher than that of external memory, but at least training and capacity expansion around internal states can bring further capability gains.

The next step is to verify whether these gains can extend beyond the current training distribution and be maintained in longer interactions and new tasks.

Back to RSI at the beginning.

Once memory nativity is achieved, it will also greatly benefit breakthroughs in continual learning and RSI.

As is well known, RSI has three pieces: memory, feedback, and verification and update mechanisms. Memory allows effective results to persist, while the latter two help determine which changes are worth entering the next round.

What MemT 1.0 is exploring now is precisely the step of carrying forward and reusing experience. It verifies that experience can guide action through native memory, filling in the first piece at the model level, and of course the most important piece.

△

If we go further, perhaps the experience accumulated in local memory can be deposited into super-memory parameters through subsequent training, and then in turn improve the model's method of managing memory.

Once this link holds, through the recorded results and methods, the system may be able to achieve cross-task intelligence explosion, which is the ultimate goal of RSI.

These are still only theoretical benefits. On the application side, memory nativity is also more conducive to enterprises completing experience accumulation.

This is certainly not wrong, but only after enterprises use it do they realize that experience accumulation is also important.

Because long-term enterprise use also involves project rules, operation sequences, and verification methods. If every time a task changes, similar constraints must be explained again and similar failures experienced repeatedly, it will be difficult for the previous investment to fully translate into subsequent returns.

Memory nativity provides a model-side carrying mechanism for this.

Under the same 128K long history, Nano's end-to-end inference speed is about 1.86 times that of the full-context solution, and within the same history length, the persistent state footprint of each trajectory is one sixty-sixth of the full-context KV Cache.

In translation, this is an immediate cost-control effect.

The state budget for long-term sessions is easier to estimate, and multiple queries can reuse the already-built state, sharing the preparation overhead.

However, to be fair, it is not necessarily true that memory nativity's task quality is definitely better than external memory, but in cases of similar quality, it is more likely to steadily reduce rework through underlying experience reuse, and this benefit is very real for enterprise operations and maintenance.

In other words, memory nativity and external memory are not mutually exclusive adversaries; through collaboration and division of labor, the two may produce greater additional benefits.

For queries about exact original text, sources, or versions, external records are still valuable; for experience invocation in subsequent execution, the role of native mechanisms can continue to be tested.

But in any case, only when effective progress can be retained and repeatedly validated in new tasks does continual self-improvement have a starting point for accumulation.

Through complete hierarchical memory management and scheduling, globally optimal read/write efficiency can be achieved, and today, the release of memory-native models has made this kind of engineering deployment and implementation possible.

The value of memory in driving continual self-improvement will also gradually become apparent in such long-term validation.

This article comes from the WeChat public account "QbitAI" (ID: QbitAI), author: Focus on Frontier Technology, published by 36Kr with authorization.

The views in this article represent only the author himself, and the 36Kr platform only provides information storage space services. +116

Great article, we need your encouragement. QbitAI invited author 2 Favorite +10 Comment. Open WeChat "Scan", open the webpage and click the share button in the upper right corner. Weibo Immersive reading Back to top Report. Join the comments. Comments are thousands, friendliness first. Log in to join the discussion. Submit comment 0/1000. You may also like these articles. DeepSeek fully open-sources its inference framework: compatible with NVIDIA and Huawei Ascend chips, single-cluster daily throughput of 155 trillion tokens, revenue of $38 per card per hour. Concentrated fulfillment, full-scale outbreak: urban NOA enters the "Qingzhou moment." So satisfying: 350,000 AIs specialize in scamming scammers, chatting and playing dumb to extract intelligence, making telecom fraud waste 17 years. Racing before Anthropic rings the bell: OpenAI sprints toward $30 billion in financing, with valuation aiming at $1.4 trillion. Is the domestic AI cost-effectiveness king about to change hands? MiniMax means business. HKUST team: helping Agents recover the "lost memories" in long conversations. Anthropic, Google, ByteDance, and Tencent enter; why are tech giants collectively rushing into AI pharmaceuticals? Meituan made a Dianping-style Skill platform DeepSkill, containing more than 370,000 skill tools. Zhipu's confusion: why has GLM, which has an excellent reputation overseas, lost to DeepSeek and Doubao in the eyes of the domestic public? QbitAI invited author

TA has not written a bio, but being reserved is also a form of expression. Published 4,100 articles.

Recent content: The three key pieces of RSI, MemT 1.0 fills in "memory" from the model layer 2 hours ago. Musk spends $10.5 billion, giving 72 million children across the United States one share of SpaceX each 7 hours ago. Inside the crash course: 4,000 cycles of validation, progress rushed and cut in half 9 hours ago. Read more content, poke here hard. View AI reviews. DeepSeek choose reliable AI, look at real reviews. View 36Kr AI review official communication community. Join consultation project review and onboarding contact 36Kr project recommendation subscription account follow.

Next article: Revisiting Dyna Robotics: after 300 days of deployment lessons, finding the robot Scaling Law.

36Kr Europe 36Kr Europe 36Kr Europe AI Product Daily Internet rumor information reporting entrance Hot recommendations

Push and interpret cutting-edge, substantive technology startup and investment news.

Primary market financial information and system service provider.

Focus on outstanding global entrepreneurs, project financing rate close to 97%, leading the industry.