Dongxi recommends a paper on generative representation learning
Dongxi NLP recommends reading the paper "Generative Representation Learning for Scalable Latent Tokenization." As introduced in the quoted post, this work proposes a diffusable representation learning method that remains scalable even when DINO/LeJEPA fail, and presents a joint scaling law for the tokenizer and diffusion model that is more compute-optimal than raw diffusion.
Topics
View original post1/ The tokenizer is a new scaling axis! We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse. We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI Thread, Paper, and Blogpost below
View original
1/ The tokenizer is a new scaling axis! We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse. We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI Thread, Paper, and Blogpost below
Body · AI translation
Recommended reading!