Luma Proposes Exact Sequence Parallelism for Packed Multimodal Masks
AI summary
Luma introduces an exact sequence parallelism method for packed multimodal masks: each layer chooses among ring attention, sparse all-to-all, or Ulysses, with numerical results consistent with a single GPU, and on up to 128 H100s it outperforms the FlashMask + Ulysses hybrid scheme in 47 of 48 cases.
Body
Exact sequence parallelism for packed multimodal masks.
It picks ring attention, a sparse all-to-all or Ulysses per layer, matches single-GPU numerics, and beats a FlashMask + Ulysses hybrid in 47 of 48 cases on up to 128 H100s.