Understand this story
Meta and MIT have published a paper reporting research findings on recurrently reusing Transformer layers: without increasing parameters, the model can achieve greater depth and outperform ordinary models. The paper states that the research trained dense models with up to 1B parameters and up to 155B tokens, and found that looping the entire stack yields the best quality at a fixed scale, while looping intermediate layers gives the highest returns when training compute is tight. Each loop adapter adds only 1.1%–4.3% parameters, whereas doubling depth requires an increase of 51.4%. Currently, this finding is presented by a single report, with no subsequent developments yet.
AI-generated from public reports
Latest development · Report
Meta and MIT Paper: Reusing Transformer Layers in a Loop Can Outperform Standard ModelsDevelopments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 2
Rohan Paul
Meta and MIT Paper: Reusing Transformer Layers in a Loop Can Outperform Standard Models
A new paper from Meta and MIT finds that reusing Transformer layers in a loop allows models to gain greater depth without adding parameters, and outperforms standard models. The research trained dense models of up to 1B parameters and up to 155B tokens, finding that looping the entire stack yields the best quality at a fixed scale, while looping intermediate layers gives the highest returns when training compute is tight. Each loop adapter adds only 1.1%–4.3% parameters, whereas doubling depth requires an increase of 51.4%.