Understand this story
On October 23, 2026, SkyRL announced that through four types of system optimizations, it extended the single-sequence training length of GLM-5.3-Flash to 1 million tokens on B200, with a peak of about 144.5 GiB, roughly 64 times that of the out-of-the-box Megatron baseline. The optimizations include: running DSA on sparse kernels, chunking by query and sharding the indexer by TP, full-layer activation recomputation and layer-input CPU offload, and removing redundant fp32 upcasting. On real book data, the 512k peak is about 122 GiB, and 1M (CP=4) is about 109 GiB; the mean absolute difference in logprob between the trainer and vLLM is about 0.04–0.07. There are currently no follow-up developments.
AI-generated from public reports
Latest development · Report
SkyRL extends GLM-5.3-Flash training context to 1 million tokensDevelopments
1 confirmed developments
SkyRL extends GLM-5.3-Flash training context to 1 million tokens
Source:Shao Meng
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 23
Shao Meng
SkyRL extends GLM-5.3-Flash training context to 1 million tokens
Through four types of system optimizations, SkyRL achieves a single-sequence training length of 1 million tokens for GLM-5.3-Flash on B200, with a peak of about 144.5 GiB, roughly 64 times that of the out-of-the-box Megatron baseline. The optimizations include running DSA on sparse kernels, chunking by query and sharding the indexer by TP, recomputing full-layer activations and CPU offloading layer inputs, and removing redundant fp32 upcasting. On real book data, the 512k peak is about 122 GiB, and 1M (CP=4) is about 109 GiB; the mean absolute difference in logprob between the trainer and vLLM is about 0.04–0.07.