Memory Tensor Open-Sources MemT 1.0 Series Memory-Native Models

In the AI world, everyone has recently fixated on three letters: RSI.

In Hinton's RSI paper, among the 22 signed authors sit Bengio, OpenAI chief scientist Pachocki, Anthropic co-founder Jack Clark... nearly half of the AI field is represented here.

How does one improvement become the starting point for the next?

Effective methods need to be retained, and the detours taken need to be identified; what carries this forward is, in fact, memory.

Memory Tensor's exploration in this area offers a completely different answer—memory-native.

What is memory-native? It means making the retention, updating, and recall of experience into capabilities that a model can learn.

When memory takes root and grows within a model, it constitutes a memory-native model.

Its latest open-source release, the MemT 1.0 series, lays out a rather concrete path:

It encodes experience into internal dynamic states, then learns through training how to write, update, and read them, and finally lets these states participate in subsequent computation via a memory branch.

Compared with external memory, not only is its capability undiminished, but its ceiling for future expansion is also higher.

This is also the first engineering validation of a memory-native model at a scale of nearly 300B parameters.

Goodness gracious—memory is invisible and intangible, yet it has genuinely become the foundation on which models grow.

From the early requirement for models to remember information to making good use of experience, the demands on memory keep escalating, pushing memory step by step from behind the scenes to center stage.

This time, MemT 1.0 directly rolls out two models:

MemT-1.0-Nano has a total of 36B parameters with 3B activated, built on the "source god" Qwen3.6-35B-A3B; MemT-1.0-Flash pushes upward, reaching 288B parameters with 17B activated, built on a further post-trained DeepSeek-V4-Flash.

What they share is that both add a memory computation pathway parallel to the original attention within selected Transformer modules.

Let's use a simple example to illustrate.

For instance, when facing the same new task, the external memory approach stuffs the original memory text, retrieved evidence, and even abstract methods back into the context, letting the model reuse experience.

The retrieval and organization process can likewise incorporate learning mechanisms.

Memory-native models like MemT 1.0 choose to go deeper inside the model:

Experience is encoded into updatable matrix states; when the model generates answers or actions, it reads the states through a dedicated memory branch, and the readout results are then fused with the original attention output to influence subsequent generation.

The difference lies in how experience is represented and through what computational process it takes effect.

But the benefits of doing this are also obvious: having reads and writes participate in training yields smoother method reuse. There is a cost, though—if precise deletion, explanation, and restoration are desired, additional mechanisms are needed.

External text is certainly easier to inspect and trace; for now, each has its own pros and cons.

To really see the difference, you have to observe how the model runs after using experience.

First, a scenario familiar to coders: modifying code.

Take a bug→locate the code→make the change; that's basically the process. But if the problem isn't reproduced at the start and boundary conditions aren't confirmed, the modification may fail and instead spawn a slew of bugs, passed down from one generation to the next.

Take SWE-bench Verified as an example; it includes an item involving Django's parse_http_date function, requiring a fix to the parsing logic for two-digit years.

The research team compared execution trajectories under three settings: no experience, textual experience, and memory-native.

Without experience, after the Agent finds the bug, it quickly starts modifying the parsing logic, and only in later steps does targeted reproduction and verification appear. In the two groups with experience, both first build reproduction tests before modifying the code.

Among them, MemT-1.0-Nano also rechecked a boundary condition after the initial modification, corrected the implementation, and performed additional verification.

It is clearly visible that experience changed the order of actions and also participated in in-task checking and correction.

Comparing on a larger sample: across 500 SWE-bench Verified tasks, each run independently three times, MemT-1.0-Nano achieved an average success rate of 69.73% at key checkpoints, versus 66.07% for the no-experience baseline and 69.13% for the textual experience group.

The proportion of tasks succeeding in all three runs rose from 54.00% without experience to 62.80%, with the textual experience group at 61.80%.

Looking next at SWE-bench Multilingual tasks under the same trigger settings, the native memory group achieved an average success rate of 63.56%, versus 60.00% for the no-experience group and 63.00% for the textual experience group.

In other words, these procedural experiences can be retained through memory-native means, performing close to explicit textual experience, with advantages showing more in average success rate and reliability across repeated runs.

More intuitively, let's look at a pixel-style maze exploration demo.

In the same maze, at the same movement speed, on the left is an Agent without memory-native capability, and on the right is MemT 1.0 with memory-native capability.

In the first round, both sides need to explore unknown routes, and both will walk into dead ends.

The difference appears after this round ends: the successful route and the forks to avoid are organized and archived, and loaded into MemT 1.0's memory to provide a reference for the next round of exploration.

By the second round, starting again from the same origin, the left side still repeatedly probes the forks, while the right side uses the experience from the previous round to avoid the dead ends it has already traveled and follows the effective route to the endpoint.

The path walked in the previous round begins to directly affect how the next round is walked—this is the concrete embodiment of why memory-native is worth doing.

Here is another information-update scenario closer to our daily office work.

The exhibition was originally scheduled for Saturday at 3 p.m., and was later officially changed to Sunday at 4 p.m. After that, someone suggested changing it to Monday, but no final decision has been made.

Remembering the last date that appeared is not enough; it also has to distinguish formal decisions from casual suggestions, and retain the content that did not change.

After multiple rounds of dialogue, MemT-1.0-Nano still successfully got it, answering the originally scheduled time and the officially revised time, and confirming that other arrangements had not changed.

In a comprehensive evaluation across 11 memory benchmarks, Flash and Nano also stood out.

Their average scores were 50.6 and 47.0 respectively, ranking first and second among the parametric memory methods participating in the evaluation, leading other models by a wide margin.

But frankly speaking, text-based memory systems as a whole still have advantages, especially in detailed recall over long histories.

What MemT 1.0 delivers today is only an initial computational implementation of the new memory-native approach, but its actual performance is already comparable to text-based memory, and as it continues to develop, its potential is also limitless.

So the question is: if experience is not kept in the task context, where does the model use it from?

This is exactly the architectural focus of MemT 1.0.

The native path provided by MemT 1.0 is to first encode relevant experience into an internal dynamic state. When generating subsequent actions, the current query reads this state through a dedicated memory branch, and then fuses the result with the original attention output.

In the specific parametric implementation, MemT 1.0 retains the base model's original computation flow and adds a native memory branch to selected Transformer modules. Broken down, it includes three layers:

With this separation, the model can not only carry new experiences, but also does not need to fine-tune the entire backbone every time; only the dynamic state corresponding to the task trajectory changes.

The read and write mechanism is also carefully designed.

When writing, the hidden state is projected to generate the keys and values of memory; erase, write, and forget gates jointly regulate the associations in the existing matrix. When reading, the current input is mapped to a memory query, the relevant information is read out, and after normalization and output gating it participates in subsequent generation.

The update mechanism uses token-by-token generation based on Gated DeltaNet-2, with stabilization processing.

(Insider tip: the author of Gated DeltaNet-2 also participated in the related discussions.)

It is triggered after each round ends, using gates to adjust existing memory associations, while within a round the state is still adjusted in token order.

Ablation experiments proved this: after replacing token-by-token updates with whole-round aggregation, Nano's average score on memory benchmarks dropped from about 47.0 to 41.0.

The six-point gap is enough to show that continuous token-level updates can better handle memory decay and promptly correct within-round associations.

But when the content of memory keeps expanding and model capacity is insufficient, what then?

At this point MemT 1.0, relying on MoE, proposes the MoS (Mixture-of-Storage) solution.

MoS expands one memory-native module into multiple memory experts, which then separately learn storage and usage routing. It adopts dense fusion: all memory experts participate in computation, and routing weights determine their respective contributions.

This is clearly different from the sparse expert routing of the base MoE.

However, structure is only the starting point.

To truly teach the model to use this pathway, the team built 34 billion tokens of memory training corpus, including 20 billion for continued pretraining and 14 billion for post-training.

The continued pretraining format is uniform: first cut long texts into consecutive segments and write them into memory, then have the model use the memory state to answer queries.

After processing each segment, the model updates the state for use by the next segment.

Post-training further aligns task results, and the data also includes three categories: memory-native language modeling data, memory storage and utilization data (such as summarization, continuation, and compression), and task data specifically strengthening long-range memory.

One point here is also worth highlighting separately: the previously frozen backbone acts as the teacher model, seeing the complete historical information, while MemT 1.0 as the student can only see its own memory state, and then off-policy distillation is used to make the student align with the teacher's distribution.

Taken together, the focus of memory-native gradually becomes clear: encoding history into a state is only the first step; how to retain, update, and read it then constitutes the whole of a memory-native model.

The value of this route still needs to be judged comprehensively in light of experience-reuse effects, resource overhead, and long-term operating conditions.

So some friends may ask again: why use memory-native?

Existing external memory solutions are already mature and have good practical cases. By comparison, memory-native has just started, and its effects are roughly similar.

Not so! Memory-native also has something unique:

It makes the representation of experience and reading/writing itself begin to become objects of model capability expansion—that is, the Memory Scaling discussed by the MemT team, namely finding a third curve of model performance growth beyond inference scaling.

Whether the original experience is useful can be seen from whether retrieval improves task results; at MemT 1.0, one can further ask: for the same experience, can it be used better with a different internal representation, update rule, and reading method?

This adds a set of variables that can be adjusted through training and architecture design to improving memory effectiveness.

Admittedly, this is still far from proving that the long-term upper limit of native memory is higher than that of external memory, but at least training and capacity expansion around internal states can bring further capability gains.

The next step is to verify whether these gains can extend beyond the current training distribution and be maintained in longer interactions and new tasks.

Back to the RSI mentioned at the beginning.

Once memory-native is achieved, it will also greatly benefit breakthroughs in continual learning and RSI.

As is well known, RSI has three pieces, including memory, feedback, and verification and update mechanisms, among which memory allows effective results to continue, and the latter two help determine which changes are worth entering the next round.

What MemT 1.0 is exploring now is precisely the step of carrying forward and reusing experience; it verifies that experience can guide action through native memory, filling in the first piece at the model level—and of course, the most important piece.

If we go further, perhaps the experience accumulated in local memory can be deposited into super-memory parameters through subsequent training, and then in turn improve the model's method of managing memory.

Once this link holds, through the recorded results and methods, the system may achieve cross-task intelligence explosion, which is the ultimate goal of RSI.

These are still only theoretical benefits; on the application side, memory-native is also more conducive to enterprises completing experience accumulation.

This is certainly not wrong, but only after enterprises use it do they realize that experience accumulation is also important.

Because long-term enterprise use also involves project rules, operation sequences, and verification methods. If every time a task is changed, similar constraints must be explained again and similar failures experienced repeatedly, it will be difficult for the previous investment to fully translate into subsequent returns.

Memory-native provides a model-side carrying mechanism for this.

Under the same 128K long history, Nano's end-to-end inference speed is about 1.86 times that of the full-context solution, and within the same history length, the persistent state footprint of each trajectory is one sixty-sixth of the full-context KV Cache.

In translation, this is an immediate cost-control effect.

The state budget for long-term sessions is easier to estimate, and multiple queries can reuse the already-built state, amortizing the preparation overhead.

However, to be fair, it is not necessarily the case that memory-native task quality is definitely better than external memory, but in cases of similar quality, it is more likely to steadily reduce rework through underlying experience reuse, and this benefit is very real for enterprise operations and maintenance.

In other words, memory-native and external memory are not mutually exclusive enemies; through collaboration and division of labor, the two may generate greater additional benefits.

For queries about exact original text, sources, or versions, external records are still valuable; for experience invocation in subsequent execution, the role of native mechanisms can continue to be tested.

But in any case, when effective progress can be retained and repeatedly withstand verification in new tasks, continual self-improvement has a starting point that can accumulate.

Through complete hierarchical memory management and scheduling, globally optimal read/write efficiency can be achieved, and today, the release of memory-native models has made this kind of engineering deployment and implementation possible.

The value of memory in driving continual self-improvement will also gradually become apparent in such long-term verification.