Silent corruption when writing Parquet files with large dictionary-encoded pages
Silent corruption when writing Parquet files with large dictionary-encoded pages: a task in LegoFlow-SWE (Harbor dataset). When writing a Parquet file with dictionary encoding and an unusually large data page size (e.g., 1 GB), the resulting file is silently corrupted. Subsequent readers — such as…
The task
When writing a Parquet file with dictionary encoding and an unusually large data page size (e.g., 1 GB), the resulting file is silently corrupted. Subsequent readers — such as Arrow C++, DuckDB, or Polars — then fail with errors like `"Unexpected end of stream"` or `"invalid dictionary indices"`.
Part of Lego-X/LegoFlow-SWE.