Accelerating hyperparameter optimization using performance prediction on a heterogeneous HPC system

About

Training and hyperparameter optimization (HPO) of deep learning-based (DL) AI models is often compute resource intensive and calls for the use of large-scale distributed resources as well as scalable and resource efficient hyperparameter search and evaluation algorithms. In this context, performance prediction emerges as a potential approach to accelerate the HPO process.

Using meta-models, referred to as performance predictors, it is possible to estimate the performance of a given configuration at a particular epoch by leveraging information from its partial learning curve. By employing performance prediction, it is possible to prioritize the training of the most promising configurations based on their predicted performance, while avoiding the need to fully train configurations with worse predicted performance. Consequently, this approach holds great potential for reducing the time and computational resources required during the HPO process.

In the context of the CoE RAISE project, this work explores novel techniques based on performance prediction to accelerate the HPO process of the particle flow reconstruction DL model known as MLPF and other NN architectures leveraging the use of High Performance Computing (HPC) resources for training the target model and the quantum annealer at JSC for training the performance predictors. Moreover, a new HPO algorithm called Swift-Hyperband, which integrates the existing Hyperband algorithm with the use of performance predictors, is proposed.

About the speaker Juan Pablo García is enrolled in the final year of a 6-year double degree program in Computer Science and Mathematics at the University of Santiago de Compostela (USC), Spain. He is currently working on his Computer Science thesis at CERN in the context of the EU funded CoE RAISE project. Juan Pablo is particularly interested in the fields where Mathematics and Computer Science intersect. Therefore, within Computer Science, he is interested in machine intelligence,…