AI News

Thinking Machines 详解 Inkling 2 Small 原生 4-bit 训练

Mira Murati更新于 11月14日 06:07

Today on Connectionism: how we trained Inkling 2 Small with a native 4-bit forward pass on @nvidia GB300s, from the first pretraining step through the final RL

AI 导读

Thinking Machines 在 Connectionism 中详细介绍了如何用原生 4-bit 前向计算训练 Inkling 2 Small,覆盖从预训练第一步到最终 RL rollout 的全流程。该训练在 NVIDIA GB300 上进行,属于整套技术栈多项改进之一,使预训练时间从数周缩短到 68 小时。Mira Murati 转发了这一内容。

推荐理由

AI 评分 62

原文披露了原生 4-bit 前向训练覆盖预训练到 RL rollout 的完整链路,以及预训练耗时从数周缩短到 68 小时的变化。

主题

标签

引用

Thinking Machines

@thinkymachines

用更少的比特进行训练速度更快,但量化误差可能会干扰学习。 今天在 Connectionism 上,我们详细介绍了如何从预训练的第一步一直到最终的 RL rollout,全程以原生 4-bit 前向计算训练 Inkling 2 Small。 https://t.co/CIMECSEpD0

查看原文

Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. https://t.co/CIMECSEpD0

查看原帖

正文

Today on Connectionism: how we trained Inkling 2 Small with a native 4-bit forward pass on @nvidia GB300s, from the first pretraining step through the final RL rollout, one of the many improvements across the stack that took pretraining from weeks to 68 hours.

来源

Mira Murati