Enhance CostBasedAutoScaler with Logarithmic Burst Scaling and Configurable Options
Enhance CostBasedAutoScaler with Logarithmic Burst Scaling and Configurable Options: a task in LegoFlow-SWE (Harbor dataset). The current autoscaler struggles to recover quickly when consumer lag per partition spikes far above the threshold. The maximum allowed increase in partitions per task is…
The task
The current autoscaler struggles to recover quickly when consumer lag per partition spikes far above the threshold. The maximum allowed increase in partitions per task is fixed to a small constant, preventing the system from temporarily adding many more tasks to burn down the backlog. We need an emergency scaling…
Part of Lego-X/LegoFlow-SWE.