Thinking Machines Lab Open-Sources the 594B-Parameter Model Inkling 2 Small
Thinking Machines Lab has open-sourced Inkling 2 Small, a 594B-parameter model, which is now available for download on Hugging Face under the Apache 2.0 license. The model uses native 4-bit forward computation throughout, from the first step of pretraining to the final RL rollout, shortening pretraining time by 8% and RL time by 20%, with an efficiency loss of about 2%. Running it requires at least 500GB of GPU memory (NVFP4) or 800GB (MXFP8); MXFP8 is only 1–2 percentage points higher than NVFP4 on evaluations. Because the same numerical format is maintained throughout training, the released 4-bit checkpoint requires no separate calibration.
Why read this
AI score 71
The original text provides the time savings and accuracy loss brought by 4-bit forward training, so readers can judge the actual trade-offs of low-bit training based on it.
Topics
View original postTraining with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. https://t.co/CIMECSEpD0
View original
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. https://t.co/CIMECSEpD0
Body
Thinking Machines Lab kept its promise and open-sourced Inkling 2 Small, a 594B-parameter model trained with 4-bit forward passes. > now available for download on HuggingFace under an Apache 2.0 licens > Training in 4-bit cut pretraining time 8% and RL time 20%, for about 2% efficiency loss.
> Running it takes at least 500GB of GPU memory in NVFP4 and at least 800GB in MXFP8, and MXFP8 gains “a very marginal 1–2 percentage points” over NVFP4 on evals.
> Because training kept the same numerics from pretraining through the final RL rollout, the released 4-bit checkpoint needed no separate calibration step.
> The lab first tried NVFP4 in both forward and backward passes, but that hurt compute efficiency too much, so only the forward pass stays in 4-bit.