Benchmark step detection does not account for measurement uncertainty
Benchmark step detection does not account for measurement uncertainty: a task in LegoFlow-SWE (Harbor dataset). When performing benchmark runs with multiple repetitions, the results include an uncertainty estimate (e.g., standard deviation or standard error). However, the step detection algorithms…
The task
When performing benchmark runs with multiple repetitions, the results include an uncertainty estimate (e.g., standard deviation or standard error). However, the step detection algorithms in ASV do not incorporate these uncertainties when identifying performance regressions. This leads to incorrect or missed step…
Part of Lego-X/LegoFlow-SWE.