Dynamic chunk sizing for large object storage uploads
Dynamic chunk sizing for large object storage uploads: a task in LegoFlow-SWE (Harbor dataset). When uploading large files to S3, GCS, or Azure blob storage, the operation fails with a part‑limit error (e.g., “Your proposed upload exceeds the maximum number of parts”). The current implementation…
The task
When uploading large files to S3, GCS, or Azure blob storage, the operation fails with a part‑limit error (e.g., “Your proposed upload exceeds the maximum number of parts”). The current implementation uses a static chunk size — default 5 MiB for S3/GCS and 4 MiB for Azure — for every part. Because the vendor limits…
Part of Lego-X/LegoFlow-SWE.