Understand this story
Google Research proposes a sleep framework for continual learning in LLMs, introducing periodic sleep phases into the model to consolidate short-term context into long-term parameters, thereby mitigating catastrophic forgetting and the static knowledge cutoff. The framework consists of two stages: memory consolidation and dreaming. The former uses MoE parameter expansion and knowledge seeding based on GKD and reinforcement learning imitation, while the latter achieves self-improvement through random expert activation, screening of synthetic samples, and RL refinement. On BABILong, this method scales nearly perfectly within 10M tokens, and achieves accuracies of 79.2%, 69.0%, and 46.1% on AIME 24/25 and HMMT 25, respectively. There are currently no follow-up developments.
AI-generated from public reports
Latest development · Report
Google Research proposes the sleep framework, enabling LLMs to retain knowledge during continual learningDevelopments
1 confirmed developments
Google Research proposes a sleep framework for continual learning of LLMs
Source:Google Research
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 28
Google ResearchFirst-partySelected
Google Research proposes the sleep framework, enabling LLMs to retain knowledge during continual learning
Google Research proposes the sleep framework, which introduces periodic sleep stages to LLMs to consolidate short-term context into long-term parameters, thereby mitigating catastrophic forgetting and static knowledge cutoff. The framework consists of two stages: memory consolidation and dreaming. The former uses MoE parameter expansion and knowledge seeding based on GKD and reinforcement learning imitation, while the latter achieves self-improvement through random expert activation, filtering synthetic samples, and RL refinement. On BABILong, this method scales nearly perfectly within 10M tokens, and achieves accuracies of 79.2%, 69.0%, and 46.1% on AIME 24/25 and HMMT 25, respectively.