Remote shuffle read performance is poor due to double compression/decompression and IPC overhead
Remote shuffle read performance is poor due to double compression/decompression and IPC overhead: a task in LegoFlow-SWE (Harbor dataset). A configuration option ballista.shuffle.remote read prefer flight should be available to control whether the optimized transport is used. When the option…
The task
A configuration option `ballista.shuffle.remote_read_prefer_flight` should be available to control whether the optimized transport is used. When the option indicates that Flight is **not** preferred, the system must fall back to the previous behaviour (no chunked raw transfer). When the optimized path is active, the…
Part of Lego-X/LegoFlow-SWE.