Ma Dongxi reads: How frontier AI teams manage
Great read!
Ma Dongxi (Dongxi NLP) shares notes from reading an article on the organizational management of frontier AI teams: the core language modeling team should always remain small, even as the company as a whole expands rapidly; pre-training planning should be few and large, while post-training can iterate relatively continuously. He also emphasizes that engineering and training must not be overly separated, and argues against releasing too many models, advocating for a fixed training schedule and having fewer but more capable models.
View original postI think it's vastly underrated how important org chart and team functionality is to building a good LLM. Over 18 months old, and still one of my favorite all time Interconnects posts. It's why we still often see orgs crumble and need to get rebuilt (e.g. Llama/xAI) and why I believe underdogs can still catch up. Having an effective org and team is worth multiple multiples in compute. I also think there isn't that much secret sauce to building models. Yes, sure there are some tricks, but most of the gains come from consistent execution and using your compute wisely.
View original
I think it's vastly underrated how important org chart and team functionality is to building a good LLM. Over 18 months old, and still one of my favorite all time Interconnects posts. It's why we still often see orgs crumble and need to get rebuilt (e.g. Llama/xAI) and why I believe underdogs can still catch up. Having an effective org and team is worth multiple multiples in compute. I also think there isn't that much secret sauce to building models. Yes, sure there are some tricks, but most of the gains come from consistent execution and using your compute wisely.
Body
Great read!
训练 frontier AI 的团队中充满着世界上最聪明的大脑，运行着巨量的算力，如何管理这样独特的组织？
随手记了很多笔记，非常受用：
1. 大团队必然滋生政治与领地意识，但语言模型的关键信息必须自下而上流动。 2. 预训练与后训练的规划逻辑不同，前者少而大，后者可相对连续迭代。 3. 核心语言建模团队应始终保持小规模，即使公司整体急剧扩张。 4.工程与训练不可过度切割，工程必须服务于下一代模型。 5. 不要发布太多模型，坚持一套固定的模型训练日程。拥有更少但更有能力的模型。
当然还有更多，推荐深度阅读！