Feature request: SparkTrials for distributed hyperparameter tuning on Apache Spark
Feature request: SparkTrials for distributed hyperparameter tuning on Apache Spark: a task in LegoFlow-SWE (Harbor dataset). Currently, hyperopt’s fmin runs optimization sequentially on a single machine , even when the user has access to a Spark cluster. The objective function is evaluated one…
The task
Currently, hyperopt’s `fmin` runs optimization **sequentially on a single machine**, even when the user has access to a Spark cluster. The objective function is evaluated one trial at a time, which becomes a bottleneck for computationally intensive hyperparameter searches.
Part of Lego-X/LegoFlow-SWE.