Google proposes a sleep framework that lets large models learn while in use without forgetting old knowledge
On October 27, Google's research team publicly released a continual learning study called the Sleep Framework, with the paper titled "Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories." The framework divides model operation into an awake phase and a sleep phase. During the sleep phase, short-term context is written into long-term parameters through parameter expansion, mixture-of-experts memory blocks, and knowledge distillation, and reasoning and generalization are improved through a dreaming step that generates synthetic samples. Experiments show that the model can handle contexts of up to about 10 million tokens, maintains near-zero performance degradation in multi-stage continual learning on unseen languages, and achieves higher accuracy on mathematical benchmarks such as AIME 24/25.
Why read this
AI score 58
Google splits model training into two phases, awake and sleep, so readers can understand how continual learning can absorb new information without overwriting old knowledge.
Topics
Body · AI translation
Farewell to the knowledge cutoff: Google teaches AI to "sleep," learning on the job without forgetting old knowledge IT Home Download the client for exclusive early access October 30, 2026, 13:51:20 From Shandong Updated October 30, 2026, 13:58:50
IT Home, October 30 news: On October 27, the Google research team published a blog post introducing a continual learning research result called the "Sleep Framework," which attempts to solve the problem of large language models having fixed knowledge after release and difficulty absorbing new information.
The research was made public by Google Research on October 27, and the related paper is titled "Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories."
Google points out that under normal circumstances, current mainstream large models enter a relatively static deployment state after completing pre-training, and their knowledge has a "cutoff date"; if they rely on in-context learning, new information is lost when the session ends. If new data is used directly for repeated fine-tuning, it easily triggers "catastrophic forgetting," and after the model learns new knowledge, its original capabilities decline significantly.
Google's approach draws on the memory consolidation mechanism in human sleep, dividing model operation into a "waking phase" and a "sleep phase." When awake, the model receives and processes input and temporarily stores information in short-term memory; after entering the sleep phase, the model pauses processing new input and instead performs memory integration and self-improvement.
Specifically, the "memory consolidation" step uses parameter expansion, mixture-of-experts memory blocks, and knowledge distillation to write fragile short-term context information into long-term parameters while avoiding overwriting existing knowledge; the "dreaming" step lets the model generate and filter synthetic training samples, repeatedly practicing through reinforcement learning to improve reasoning and generalization capabilities.
Experiments show that the framework outperforms various baseline methods in tasks such as continual translation, long-context understanding, factual knowledge absorption, and mathematical reasoning: the model can handle contexts of up to about 10 million tokens and maintains near-zero performance degradation in multi-stage continual learning on unseen languages; it also achieves higher accuracy on mathematical benchmarks such as AIME 24/25.
"Special statement: The above work content (including videos, pictures, and audio if any) is uploaded and published by users of Dafeng Hao, a self-media platform under Phoenix Net. This platform only provides information storage space services.
Notice: The content above (including the videos, pictures and audios if any) is uploaded and posted by the user of Dafeng Hao, which is a social media platform and merely provides information storage space services."