Affiliations: [a] Computer Science Department, College of Computers and Informatics, Zagazig University, Zagazig, Egypt
| [b] College of Computer Science and Electronic Engineering, Hunan University, Changsha, China
Correspondence:
[*]
Corresponding author. Ahmad Salah, E-mail: ahmad@hnu.edu.cn.
Abstract: Hyperparameter optimization is a crucial step in the implementation of any machine learning model. This optimization process includes regularly modifying the hyperparameter values of the model in order to minimize the testing error. A deep neural learning model hyperparameter optimization process includes optimizing both the model parameters and architecture. Optimizing a model’s parameters involves deciding the values of parameters, such as learning rate and batch size. Optimizing architectural hyperparameters includes deciding the shape of the deep neural learning model, i.e., the number of layers of individual types and the number of neurons in a certain layer. The state-of-the-art hyperparameter optimization methods don’t optimize the position of the hyperparameter within the model architecture. In this work, we study the effect of changing a hyperparameter within the deep learning model architecture. Thus, we propose an architectural position optimization (ArchPosOpt) method for model architectural hyperparameter optimization. ArchPosOpt extends three different hyperparameter optimization techniques, namely grid search, random search, and Tree-structured Parzen Estimator (TPE), to include a new dimension of hyperparameter optimization problem – the hyperparameter position. We show through a set of experiments that the position of the hyperparameters does matter for model performance as well as the hyperparameter values.
Keywords: Deep neural networks, hyperparameter optimization, CNN, architectural optimization, hyperparameter position
