When using file-based IOs like AvroIO, TextIO, or TFRecordIO with a partitioned PCollection that has a large…
When using file-based IOs like AvroIO, TextIO, or TFRecordIO with a partitioned PCollection that has a large…: a task in LegoFlow-SWE (Harbor dataset). The underlying WriteFiles transform already provides a configuration option withMaxNumWritersPerBundle that caps the number of writers opened per…
The task
The underlying `WriteFiles` transform already provides a configuration option `withMaxNumWritersPerBundle` that caps the number of writers opened per bundle, spilling additional partitions to shuffle to reduce memory pressure. However, this option is not exposed on the higher-level `Write` builders returned by…
Part of Lego-X/LegoFlow-SWE.