Add a corpus shuffling-and-splitting utility for streaming data loading
Add a corpus shuffling-and-splitting utility for streaming data loading: a task in MiMo-V2.6-RL-oss: Agentic RL Environments (MiMo RL release). When we train a language model from scratch we feed the trainer with a StreamingDataSilo, which reads training data lazily from disk instead of loading…
The task
When we train a language model from scratch we feed the trainer with a StreamingDataSilo, which reads training data lazily from disk instead of loading everything into memory. To get good randomization across epochs with that streaming setup it helps to pre-process the raw…
Part of XiaomiMiMo/MiMo-V2.6-RL-oss.