Parcae:用稳定循环模型以更少参数实现更高性能
Together AI 发布稳定循环语言模型架构 Parcae,通过反复使用相同层来增加计算量,而不扩大参数规模。实验中,Parcae 的验证困惑度较参数和数据匹配的 RDM 最多降低 6.3%,770M 模型的下游表现接近参数量两倍的 Transformer;训练代码和模型已发布。
to scale quality by increasing recurrence rather than purely scaling data, opening up an efficient frontier for training memory-constrained on-device models. Traditional scaling laws tell us that to achieve the best performance, we need to scale FLOPs, often with more parameters or data. But as models move to the edge and inference costs skyrocket, we wonder: Can we scale quality without inflating memory footprint? To that end, we’ve been exploring looped architectures, models that increase com
匹配字段正文








