Google Research releases SLEEP: using offline sleep cycles to mitigate catastrophic forgetting in large models

Google Research has just released "SLEEP": giving large models a periodic "offline memory consolidation" cycle to mitigate catastrophic forgetting.

What SLEEP aims to solve: after a model is deployed, its knowledge is essentially frozen; hard-tuning parameters easily causes catastrophic forgetting, while relying solely on in-context learning means it forgets once the window is exceeded. The approach is to have the LLM periodically switch between "awake interaction" and "sleep consolidation" under a fixed architecture and established training rules: first write the fragile information in short-term context into expandable long-term parameters; then in the "dreaming" phase, use synthetic data for self-improvement. https://t.co/VJ6VCRbpRK

Where the problem comes from Pretraining + in-context learning (ICL) is powerful, but after deployment it remains approximately static, with a knowledge cutoff date. Continuously changing parameters often washes out old capabilities (catastrophic forgetting); relying only on ICL means knowledge stays in the current session, and once the session ends, short-term information is gone.

Awake vs. sleep · Awake (Active / wake): normally receives and processes input; temporary information is placed in fast, short-term memory components. · Sleep: pauses new input and shifts compute toward memory consolidation and self-improvement. The offline phase unlocks pre-reserved MoE memory blocks and updates weights using practice samples generated by the model itself. The terminology is borrowed from neuroscience, but the implementation is still automated post-training on a fixed architecture, without changing the core code or design.

Two steps within sleep 1. Memory consolidation (analogous to NREM / slow-wave sleep) · Parameter expansion and continuous memory hierarchy: dynamic expansion during sleep. Combining MoE with the Hope architecture mentioned in the text, newly unlocked low-rank experts are written into more stable, lower-frequency memory blocks, minimizing changes to old parameters. · Knowledge seeding / upward distillation: migrate fragile short-term traces from high-frequency layers (such as context memory) into newly expanded long-term parameters. Technically, this uses generalized knowledge distillation (GKD) plus reinforcement-learning-based imitation: on one hand, on-policy distillation (the student learns from token-by-token feedback of the teacher distribution on self-sampled sequences); on the other hand, Learning to Imitate (LTI), which scores prefix continuations by similarity measures such as semantics and edit distance (Levenshtein). · Synaptic pruning: after consolidation is complete, reset high-frequency temporary parameters to free up capacity for the next awake session.

2. Dream-style self-improvement (analogous to REM) · When generating synthetic data, MoE routing randomly activates unconventional experts, encouraging cross-domain mixing and uncovering latent patterns. · Use gradient-based data filtering to keep the "dream" samples most likely to improve overall capability. · Then perform supervised fine-tuning on the selected samples and check whether downstream tasks actually improve.

Theoretical side Consolidation is viewed as structured low-rank compression: rather than copying short-term traces verbatim into parameters, it selects directions that both align with the model's own on-policy behavior and preserve semantics and task-related rewards; parameter expansion provides "write space," reducing overwriting of old knowledge. Dreaming is described as improving the geometry of learning: random experts broaden gradient directions, and gradient filtering then emphasizes useful updates, making optimization more balanced and stable.

Key results reported in the text · Continual translation: in multi-stage evaluations on unseen languages (such as Manchu and Kalamang), SLEEP shows near-zero degradation; baselines such as ICL, Cartridges, and supervised fine-tuning (SFT) forget more heavily. · Long context: on BABILong, it is claimed to scale almost perfectly to about 10 million tokens, and sleep-phase consolidation is used to compress long documents into parametric memory, compared with GPT-4, Titans, and others. · Fact injection: under SQuAD / continual pretraining settings, about 48.9% for a single task and about 46.2% for CPT, outperforming the base model, SFT, SEAL, and other reported figures. · Mathematical reasoning: the sleep-enhanced base model achieves about 79.2% / 69.0% / 46.1% on AIME 24/25 and HMMT 25, respectively, outperforming SFT, GRPO, OPSD, and others.