Understand this story
Luma proposes an exact sequence parallelism method for packed multimodal masking. At each layer, the method chooses among ring attention, sparse all-to-all, or Ulysses, and its numerical results are consistent with those on a single GPU. On up to 128 H100 GPUs, the method outperforms the FlashMask + Ulysses hybrid scheme in 47 of 48 cases. There are currently no follow-up developments.
AI-generated from public reports
Latest development · Report
Luma Proposes Exact Sequence Parallelism for Packed Multimodal MasksDevelopments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 20
Luma AI
Luma Proposes Exact Sequence Parallelism for Packed Multimodal Masks
Luma introduces an exact sequence parallelism method for packed multimodal masks: each layer chooses among ring attention, sparse all-to-all, or Ulysses, with numerical results consistent with a single GPU, and on up to 128 H100s it outperforms the FlashMask + Ulysses hybrid scheme in 47 of 48 cases.