谢赛宁：RAE 与 MoE 协同或破解 DiT 扩展难题
another scientific exploration from @TongPetersb, @DavidJFan, and @__JohnNguyen__ that might teach you something new, even if you’re in a frontier lab
来自 @TongPetersb、@DavidJFan 和 @__JohnNguyen__ 的又一次科学探索，即使你在前沿实验室，也可能学到新东西。 这里有很多有趣的观察，但我只强调一点： - 试图用 MoE 扩展 DiT 大多徒劳无功，这算是业界公开的秘密。 - RAE 与 MoE 之间意外却又直觉的协同，或许能改变这一点。
查看原帖Train Beyond Language. We bet on the visual world as the critical next step alongside and beyond language modeling. So, we studied building foundation models from scratch with vision. We share our exploration: visual representations, data, world modeling, architecture, and scaling behavior! [1/9]
正文
当前显示原文，中文全文译文尚未就绪。
another scientific exploration from @TongPetersb, @DavidJFan, and @__JohnNguyen__ that might teach you something new, even if you’re in a frontier lab
lots of interesting observations here, but I’ll highlight just one: - it’s kind of an open industry secret that trying to scale DiTs with MoE has mostly been fruitless. - the unexpected, yet intuitive, synergy between RAE and MoE might actually change that.