Support Sequence Parallelism for Training Very Long Sequences
Support Sequence Parallelism for Training Very Long Sequences: a task in LegoFlow-SWE (Harbor dataset). When training models with extremely large numbers of tokens (e.g., 1,280,000 or more) using flash attention, a RuntimeError is raised during the forward pass, even when sufficient GPU memory is…
The task
When training models with extremely large numbers of tokens (e.g., 1,280,000 or more) using flash attention, a `RuntimeError` is raised during the forward pass, even when sufficient GPU memory is available. The failure occurs inside `flash_attn_varlen_qkvpacked_func` due to internal size limitations. This prevents…
Part of Lego-X/LegoFlow-SWE.