Support dynamic (variable-length) batch sizes in BatchSamplerShard, including the even batches=True padding…
Support dynamic (variable-length) batch sizes in BatchSamplerShard, including the even batches=True padding…: a task in HF ML Bench v0 (Harbor dataset). Extend the distributed batch-sampler sharder so that it correctly handles a wrapped batch sampler that yields variable-length batches — the case…
The task
Extend the distributed batch-sampler sharder so that it correctly handles a wrapped batch sampler that yields **variable-length batches** — the case where the batch sampler has no fixed `batch_size` (i.e., `getattr(batch_sampler, "batch_size", None) is None`), and each batch it produces is a list of indices of…
Part of AdithyaSK/HF_ML_Bench_v0.