Ma Dongxi reads: How frontier AI teams manage
Ma Dongxi (Dongxi NLP) shares notes from reading an article on the organizational management of frontier AI teams: the core language modeling team should always remain small, even as the company as a whole expands rapidly; pre-training planning should be few and large, while post-training can iterate relatively continuously. He also emphasizes that engineering and training must not be overly separated, and argues against releasing too many models, advocating for a fixed training schedule and having fewer but more capable models.
View original postI think it's vastly underrated how important org chart and team functionality is to building a good LLM. Over 18 months old, and still one of my favorite all time Interconnects posts. It's why we still often see orgs crumble and need to get rebuilt (e.g. Llama/xAI) and why I believe underdogs can still catch up. Having an effective org and team is worth multiple multiples in compute. I also think there isn't that much secret sauce to building models. Yes, sure there are some tricks, but most of the gains come from consistent execution and using your compute wisely.
View original
I think it's vastly underrated how important org chart and team functionality is to building a good LLM. Over 18 months old, and still one of my favorite all time Interconnects posts. It's why we still often see orgs crumble and need to get rebuilt (e.g. Llama/xAI) and why I believe underdogs can still catch up. Having an effective org and team is worth multiple multiples in compute. I also think there isn't that much secret sauce to building models. Yes, sure there are some tricks, but most of the gains come from consistent execution and using your compute wisely.
Body · AI translation
Great read!
The teams training frontier AI are filled with the world's brightest minds and run enormous amounts of compute—how do you manage such a unique organization?
I jotted down a lot of notes, and found them very useful:
1. Large teams inevitably breed politics and territoriality, but key information about language models must flow bottom-up. 2. The planning logic for pretraining and post-training differs: the former is few and large, while the latter can iterate relatively continuously. 3. The core language modeling team should always remain small, even as the company as a whole expands rapidly. 4. Engineering and training must not be overly separated; engineering must serve the next generation of models. 5. Don't release too many models; stick to a fixed model training schedule. Have fewer but more capable models.
Of course there's more—deep reading recommended!