Modern machine learning algorithms crucially rely on several design decisions
to achieve strong performance, making the problem of Hyperparameter
Optimization (HPO) more important than ever. Here, we combine the advantages of
the popular bandit-based HPO method Hyperband (HB) and the evolutionary search
approach of Differential Evolution (DE) to yield a new HPO method which we call
DEHB. Comprehensive results on a very broad range of HPO problems, as well as a
wide range of tabular benchmarks from neural architecture search, demonstrate
that DEHB achieves strong performance far more robustly than all previous HPO
methods we are aware of, especially for high-dimensional problems with discrete
input dimensions. For example, DEHB is up to 1000x faster than random search.
It is also efficient in computational time, conceptually simple and easy to
implement, positioning it well to become a new default HPO method.

DEHB is a new Hyperparameter Optimization method that combines the advantages of Hyperband and Differential Evolution, and achieves strong performance far more robustly on a broad range of HPO problems and tabular benchmarks, especially for high-dimensional problems with discrete input dimensions.