Nathan Lambert: Organizational structure is severely underestimated for building good LLMs
Nathan Lambert believes that the importance of organizational structure and team collaboration for building excellent LLMs is severely underestimated, and that the value of an efficient organization and team is equivalent to several times the compute. Citing team reorganizations at Llama, xAI, and other teams as examples, he explains that after an organizational collapse, things need to be rebuilt, which is also why he believes latecomers can still catch up. He also states that there are not many unique secrets to building models, and that most of the gains come from sustained execution and wise use of compute.
Body
I think it's vastly underrated how important org chart and team functionality is to building a good LLM. Over 18 months old, and still one of my favorite all time Interconnects posts. It's why we still often see orgs crumble and need to get rebuilt (e.g. Llama/xAI) and why I believe underdogs can still catch up. Having an effective org and team is worth multiple multiples in compute.
I also think there isn't that much secret sauce to building models. Yes, sure there are some tricks, but most of the gains come from consistent execution and using your compute wisely.