Mirendil Releases Fern, Its First Model Pretrained from Scratch, Supporting 10M Token Context
Mirendil released Fern, its first model pretrained from scratch, with a context of up to 10M tokens, equivalent to roughly 500,000 lines of code (Megatron-LM scale) or several days of an agent's work trajectory. The first version, Fern 0.1, underwent only a short round of post-training; its architecture, training, and serving were all designed for a 10M context, and next-token metrics continue to improve as context length grows. It targets high-throughput work such as auditing data, reading training logs, and reading through agent trajectories, and lists six scenarios including work on larger repositories, agentic coding, AI research code, and long-log debugging.
Why read this
AI score 60
The original text provides the specific capacity conversion for the 10M token context and six high-throughput scenarios, making it easier to judge the applicable boundaries of ultra-long-context models.
Topics
View original postIntroducing Fern, Mirendil’s first model: pretrained from scratch, with up to 10M tokens of context! That’s ~0.5M lines of code (roughly Megatron-LM scale) or days of an agent’s work, all in context at once. Its first version, Fern 0.1, had a short round of post-training 🚢 https://t.co/ClnV1Z3yzN
View original
Introducing Fern, Mirendil’s first model: pretrained from scratch, with up to 10M tokens of context! That’s ~0.5M lines of code (roughly Megatron-LM scale) or days of an agent’s work, all in context at once. Its first version, Fern 0.1, had a short round of post-training 🚢 https://t.co/ClnV1Z3yzN
Body · AI translation
Mirendil releases Fern, its first model pretrained from scratch: treating 10M tokens as another kind of "compute," serving "people who build AI" @mirendil https://t.co/8Jwai8sGDu
What 10M tokens can do: ~23 MB of code / ~500K lines, Megatron-LM scale, or fit an entire "several days of an agent working" trajectory into context at once. Version 0.1 only did short-turn post-training; Mirendil's narrative is that "parameters and context are two paths for spending compute," with architecture / training / serving all designed around 10M; next-token metrics still improve as context grows longer, and it claims prediction quality holds all the way to 10M. Aimed at high-throughput work like auditing data, reading training logs, and reading through agent trajectories; 0.1 only did short-turn post-training.
Six use cases · Work on larger repositories · agentic coding (run commands, read output, until tests pass) · AI research code (training loops, data pipelines, evaluation) · Debugging from long logs/stack traces · Reading through entire agent/training trajectories and explaining causality · Scanning whole pretraining shards for duplicates, benchmark contamination, and junk data