Issue: Performance overhead during dataset open due to eager layout copy
Issue: Performance overhead during dataset open due to eager layout copy: a task in LegoFlow-SWE (Harbor dataset). Opening many small datasets is slower than necessary, even when compiled with NDEBUG . Benchmarks that create and open a large number of datasets (e.g., size = 10000 ) show a…
The task
Opening many small datasets is slower than necessary, even when compiled with `NDEBUG`. Benchmarks that create and open a large number of datasets (e.g., `size = 10000`) show a measurable overhead in the open path. Profiling reveals that the dataset layout is immediately copied to the internal **Dataset Creation…
Part of Lego-X/LegoFlow-SWE.