Introduce LinkedBuffer to avoid humongous allocations in S3 streaming
Introduce LinkedBuffer to avoid humongous allocations in S3 streaming: a task in LegoFlow-SWE (Harbor dataset). When the S3 filesystem writer is used with multiple threads, it allocates large (default 32 MB) contiguous byte arrays for buffering. With the G1 garbage collector, these arrays become…
The task
When the S3 filesystem writer is used with multiple threads, it allocates large (default 32 MB) contiguous byte arrays for buffering. With the G1 garbage collector, these arrays become **humongous objects** (slightly larger than a region, causing 50 % wasted heap in two regions). This leads to severe heap…
Part of Lego-X/LegoFlow-SWE.